Networks and distributed storage allow data and storage space to be shared between devices located anywhere a connection is available. These implementations may range from a single machine offering a shared drive over a home network to an enterprise-class cloud storage array with multiple copies of data distributed throughout the world. Larger implementations may incorporate Network Attached Storage (NAS) devices, Storage Area Network (SAN) devices, and other configurations of storage elements and controllers in order to provide data and manage its flow. Improvements in distributed storage have given rise to a cycle where applications demand increasing amounts of data delivered with reduced latency, greater reliability, and greater throughput.
In view of these trends, there is a tendency to group this data into increasingly larger volumes (logical partitions) spanning multiple drives. However, larger volumes place more data at risk should one of the underlying storage devices fail. Even though storage devices have become more reliable, they are not infallible. To reduce the risk of data loss in the event hardware becomes inaccessible, RAID (Redundant Array of Independent/Inexpensive disks) and other groupings offer redundancy protection such as parity or mirroring. Data lost due to a failed device can be recovered from the parity or mirrored data provided a sufficient number of devices within the group are still functional.
In architectures that offer redundancy, the group or array may continue to function even with one or more inaccessible devices. However, there is no guarantee that changes to the data will be written to the inaccessible devices. Thus, even if an inaccessible device comes back online, it may be out of sync with the remainder of the group. One solution is to reconstruct the entire dataset of the now-accessible device and to store a clean copy on the device. This is a non-trivial task, and in some applications, a rebuild in this manner can take hours or days. Accordingly, while existing techniques for maintaining a group of storage devices have been generally adequate, the techniques described herein dramatically reduce rebuild time, provide improved write performance in the event of a device failure, and do so with less burden on the storage system.